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Our  project  sought  to  understand  the  changes  in  behavior  and  brain  that  accompany  the  development  of 
expertise  in  the  telecontrol  of  tools  and  devices.  Particular  emphasis  was  placed  on  the  control  of  multi-step 
actions  in  which  ongoing  behavior  must  be  adjusted  in  anticipation  of  forthcoming  task  demands.  These 
anticipatory  adjustments  to  behavior  reflect  the  use  of  internal  representations.  We  used  functional  magnetic 
resonance  imaging  (fMRI)  and  behavioral  measures  to  investigate  the  changes  that  occur  in  these  internal 
representations  as  operators  develop  expertise  in  controlling  manual  tools  and  a  remotely  located  robotic  arm 
for  goal-oriented  reaching,  grasping  and  object  manipulation.  Results  of  this  work  may  have  widespread 
implications  for  the  increasingly  commonplace  situations  in  modem  combat  and  telemedical  environments 
where  individuals  must  plan  and  operate  a  wide  variety  of  devices  remotely,  under  highly  variable  contexts,  and 
with  limited  sensory  feedback. 

As  summarized  below,  and  in  the  attached  figures,  we  have  pursued  four  general  objectives  during  the  past  year. 

Objective  1:  Characterization  of  the  behavioral  and  neural  signatures  of  anticipatory  movement  selection.  The 
way  that  we  choose  to  grasp  an  object  (e.g.,  over-  vs.  under-hand)  depends  on  sensory  information  concerning 
the  state  of  the  body  (e.g.,  current  posture)  and  the  target  object  (e.g.,  location,  orientation),  as  well  as 
anticipation  of  forthcoming  task  demands  (e.g.,  intended  rotation  of  the  target  object).  For  instance,  when 
grasping  a  handle  with  the  intention  to  rotate  it,  participants  may  select  a  less-comfortable  grip  in  order  to  end  in 
a  more  comfortable  posture  (Rosenbaum  &  Jorgensen,  1990).  Are  such  anticipatory  effects,  evident  in  multi- 
step  actions,  the  result  of  feed  forward  (predictive)  mechanisms  that  contribute  more  generally  to  motor  control? 
Event-related  fMRI  was  used  to  ask  this  question  in  15  healthy,  right-handed  adults.  Participants  were  required 
to  select  the  most  comfortable  way  (over-  vs.  under-hand)  to  grasp  a  handle  using  either  hand  with  the  intention 
of  simply  grasping  the  handle  (NO-ROTATION)  or  rotating  it  90°  clockwise  or  counterclockwise 
(ROTATION). .  Responses  indicated  which  end  of  the  handle  the  thumb  would  be  on,  and  no  overt  movements 
were  made.  As  expected  if  they  were  accurately  anticipating  forthcoming  task  demands  participants  were  more 
likely  to  adopting  an  underhand  grip  in  ROTATION  vs,  NO-ROTATION  condition.  Consistent  with  earlier 
work,  grip  selection  planning  in  the  NO-ROTATION  condition  was  associated  with  increased  activity  in 
inferior  frontal  and  dorsal  premotor  cortices,  pre-supplementary  motor  area,  intraparietal  sulci  and  lateral 
cerebellum.  As  shown  in  Figure  1,  planning  in  the  ROTATION  condition  evoked  additional  increases  in 
activity  within  these  very  same  brain  regions.  These  findings  are  consistent  with  the  hypothesis  that  the  same 
brain  regions  underlie  anticipatory  planning  motor  planning  across  multiple  timescales.  These  results  are 
currently  under  revision  for  resubmission  to  the  Journal  of  Neurophysiology. 

Objective  2:  Development  of  an  MRI-compatible  system  to  investigate  internal  representations  underlying 
telecontrol  of  a  robotic  arm.  Our  first  pass  at  a  glove  controller  mapped  individual  cyber-glove  sensors  to  servos 
on  the  robot  (Figure  2).  Controlling  the  robot  this  way  was  confusing  and  difficult  to  master.  Furthermore, 
once  the  robot  was  in  the  desired  position  the  participant’s  hand  had  to  be  held  perfectly  steady  while,  for 
example,  opening  or  closing  the  gripper  or  the  robot  would  move. 

We  came  up  with  two  improvements  to  the  initial  system  and  have  continued  to  test  these.  The  first  was  to 
implement  a  “joystick”  mode  in  the  glove  controller  software.  In  this  mode,  only  the  ends  of  the  range  of 
movement  for  a  joint  associated  with  a  sensor  cause  the  robot  to  move.  The  center  of  the  joint  movement  range 
is  a  “dead  zone”,  allowing  the  participant  to  maintain  her  hand  in  a  neutral  position  while  keeping  the  robot 
from  moving.  The  second  improvement  addresses  the  difficulty  in  moving  the  robot  by  controlling  individual 
joints.  Instead,  we  implemented  an  inverse  kinematics  solution  where  glove  movements  control  the  position  of 
the  robot’s  end  effector  in  space.  The  robot’s  servo  values  are  then  back-calculated  from  the  desired  end 
effector  position.  Since  one  of  our  aims  is  to  allow  live  control  of  the  robot,  we  needed  to  make  our  calculations 
as  fast  as  possible.  Therefore  we  implemented  a  geometric  inverse  kinematics  solution  rather  than  an  iterative 
one.  For  this  calculation  we  had  the  option  of  letting  the  participant  use  Cartesian,  cylindrical,  or  spherical 
coordinates.  While  Cartesian  coordinates  would  probably  be  more  familiar  to  the  average  participant,  the  types 


well  as  the  calculations  involved,  we  decided  to  keep  the  gripper  parallel  to  the  ground.  With  this  added 
constraint,  the  cylindrical  coordinate  system  ended  up  making  the  most  sense.  We  are  presently  pilot  testing  a 
learning  paradigm  that  will  enable  us  to  track  neural  correlates  of  the  development  of  telecontrol  expertise  using 
this  system. 

Objective  3:  Parieto-premotor  contributions  to  the  planning  and/or  execution  of  reaching  or  grasping 
movements  studied  in  humans  with  fMRI.  There  is  considerable  evidence  implicating  the  anterior  intraparietal 
area  (AIP)  and  ventral  premotor  cortex  (F5)  of  macaques  in  sensori-motor  transformations  for  grasp.  Studies  of 
visually-guided  grasping  in  humans  consistently  detect  activation  near  the  junction  of  the  anterior  intraparietal 
and  postcentral  sulci  (alPS).  Interesting  exceptions  include  tasks  in  which  participants  grasp  under  open-loop 
conditions  (i.e.,  without  visual  feedback,  Binkofski  et  al.,  1999),  and  when  they  plan  grasping  actions  while 
remaining  still  (Jacobs  et  al.,  2009).  These  findings  raise  the  possibility  that  vPMC  may  play  a  greater  role  in 
feed-forward  (predictive)  control,  while  alPS  is  more  involved  in  feedback  processing.  To  address  this 
question,  we  used  a  rapid,  event-related  fMRI  paradigm  to  distinguish  activity  involved  in  the  planning  vs. 
execution  phases  of  reaching  and  grasping  movements  in  18  healthy  right-handed  adults. 

At  the  onset  of  each  trial  an  instructional  cue  indicated  whether  the  forthcoming  movement  would  involve 
reaching  or  grasping.  This  was  followed  by  a  variable  duration  delay  period  (3  -  4s).  Trials  were  separated  by 
a  fixed  duration  intertrial  interval  (0.5s),  and  one  third  of  trials  within  the  run  consisted  of  null  events.  Onset  of 
the  visual  display  served  as  the  movement  cue.  Subjects  then  reached  toward  or  grasped  a  25  x  25  x  50mm 
wooden  block  with  their  right  hands.  In  the  grasp  condition,  the  block  was  picked  up,  transported  laterally  and 
placed  in  a  50mm  diameter,  circular  opening.  In  the  reach  condition,  the  fingertips  were  kept  in  contact  with 
each  other  as  the  subject  touched  the  top  of  the  block  and  then  moved  the  hand  over  the  hole.  As  expected,  the 
comparison  of  grasp  execution  vs.  reach  execution  yielded  increased  activity  in  alPS  (Figure  3).  Importantly, 
activity  within  this  functionally-defined  region  of  interest  only  shows  increased  activity  for  reach  and  grasp 
execution  and  not  for  planning.  By  contrast,  more  caudal  angular  gyrus  (ANG)  showed  significant  increases  in 
activity  for  planning  grasping  or  reaching,  but  not  for  execution  of  either  action  (Figure  4).  These  findings 
suggest  that  the  human  alPS  may  be  more  heavily  involved  in  sensorimotor  control  of  grasp,  while  more  caudal 
regions  of  posterior  parietal  cortex  participate  in  premovement,  feedforward  planning  of  both  grasp  and  reach. 
This  work  was  presented  at  the  2010  Meeting  of  the  Society  for  Neuroscience  and  a  manuscript  is  in  preparation 
for  submission  to  a  peer-reviewed  journal. 

Objective  4:  fMRI  Experiment  1:  Responses  of  the  parietofrontal  grasp  network  to  control  of  telecontrol  robot 
grasping.  As  summarized  in  Objective  3  above,  there  is  a  network  of  areas  in  parietal  and  premotor  cortex  that 
are  involved  in  the  internal  representation  of  grasping.  Importantly,  we  showed  that  these  same  areas  come  to 
code  grasping  with  a  novel  handheld  tool  once  operators  have  achieved  a  level  of  expertise.  Put  differently,  it 
appears  that  these  regions  code  grasping  independent  of  the  effector  involved.  In  this  initial  experiment  with  the 
robotic  arm,  we  ask  whether,  following  training,  this  “grasp”  network  comes  to  also  represent  grasping  with  a 
telecontrolled  robotic  arm.  To  keep  matters  simple,  the  operator  was  trained  to  press  one  button  to  control  the 
position  of  the  robot  (i.e.,  reach),  and  another  to  control  movements  of  the  gripper  (i.e.,  grasp).  We  reasoned 
that  if  this  parieto-premotor  circuit  is  involved  in  the  control  of  grasping  independent  of  the  effector  involved, 
then  telecontrolled  grasping  will  increase  activity  selectively  within  these  regions.  As  shown  in  Figure  5,  this  is 
what  we  found.  After  developing  expertise  in  controlling  the  robotic  arm,  participants  show  engagement  of 
posterior  parietal  (green  arrows)  and  premotor  (blue  arrows)  regions  normally  involved  in  the  control  of  manual 
reach  (left)  and  grasp  (right).  This  even  when  the  hand  movements  used  to  control  the  robot  are  simple  button 
presses.  This  suggests  that  learning  to  control  remote  devices  under  teleoperational  conditions  is  accomplished 
within  the  same  circuits  that  underlie  comparable  goal  directed  actions  involving  one’s  own  body.  This  is 
consistent  with  what  was  found  earlier  in  this  project  for  use  of  handheld  mechanical  tools.  We  believe  that  this 
information  may  have  practical  implications  for  development  of  training  protocols  for  telecontrol  and  for 
enhancement  of  brain-controlled  interfaces.  A  manuscript  is  in  preparation  for  submission  to  a  peer-reviewed 
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